Video encoding with compression enables storing, transmitting, and processing of audio-visual information with fewer storage, network, and processor resources. The most widely used video compression standards include MPEG-1 for storage and retrieval of moving pictures, MPEG-2 for digital television, and MPEG-4 and H.263 for low-bit rate video communications, see ISO/IEC 11172-2:1991. “Coding of moving pictures and associated audio for digital storage media at up to about 1.5 Mbps,” ISO/IEC 13818-2:1994, “Information technology—generic coding of moving pictures and associated audio,” ISO/IEC 14496-2:1999, “Information technology—coding of audio/visual objects,” and ITU-T, “Video Coding for Low Bitrate Communication,” Recommendation H.263, March 1996.
Most video compression schemes are designed to remove spatial redundancies within a single video frame and temporal redundancies between a sequence of video frames. A discrete cosine transform (DCT) is frequently used to compact signal energy and reduce the spatial redundancies, while motion compensation is used to minimize the temporal redundancies. Video frames are usually subdivided into macroblocks, e.g., 16×16 blocks, and the typical coding operations, such as motion compensation and DCT, are performed on a macroblock basis.
Frames can be encoded in two ways: (1) as intra-frames, e.g., I-frames, in which the entire frame is coded independently of any other frame, or (2) as inter-frames, e.g., P-frames, which employ motion compensation techniques and rely on the prediction of neighboring frames. Inter-coded frames may contain macroblocks that are intra-coded.
Compressed video is extremely sensitive to transmission errors due to the predictive nature in which the video signal is encoded. Errors introduced into the bitstream by channel impairments can result in very severe artifacts in the decoded video. Error recovery techniques for general data communication have been developed. For example, retransmission techniques aim for lossless delivery. However, such techniques cannot be applied to streaming video transmission due to delay constraints.
Error-resilient encoding techniques developed especially for videos are necessary for robust and reliable video transmission over error prone transport channels. The primary reason is that with video, even if all the bits of a frame are received correctly, it is still possible for the frame to be corrupt due to error propagation from a previously decoded frame. Motion compensation removes inter-frame temporal redundancy by encoding only a prediction error between a current frame and previously encoded frames, i.e., reference frames. Therefore, the loss of a reference frame can introduce errors that propagate into subsequent derived frames.
A fundamental problem in designing error-resilient video encoding schemes is that there exist contradictory requirements between compressing efficiency and error resilience. When the transport channel is lossy, there are two types of distortion observed at the decoder. The first is the encoding distortion, which is caused by quantization noise introduced during compression. The second is the distortion due to transmission errors. The total distortion includes the effects from both encoding distortion and transmission distortion and is referred to as end-to-end distortion.
Given an available bit rate (bandwidth), high compression efficiency requires one to minimize the encoding distortion by removing redundancy in the encoded bitstream. On the other hand, high error resilience requires reduction in transmission distortion by adding redundancy into the encoded bitstream.
Ideally, the video encoding scheme should balance the compression and error-resilience such that the end-to-end distortion is minimized. This must also be done under current channel characteristics, which account for the available bit rate on the channel and the channel error rate. It should be noted that most conventional video codecs do not consider methods for dynamically varying the amount of error resilience in the coded bitstream, but only focus on minimizing the coding distortion given the available bit rate.
In U.S. Pat. No. 6,317,462, “Method and apparatus for transmitting MPEG video over the internet,” issued on Nov. 13, 2001 to Boyce et al., a coded bitstream is split into two partitions: a high-priority partition and a low-priority partition. A systematic forward error erasure/correction (FEC) code, such as a Reed Solomon (n, k) code, is then applied to bits in the high priority partition. Boyce focuses on protecting important information at the bit level. However, error propagation effects are not addressed.
One useful technique to mitigate error propagation effects is to force some macroblocks to be coded as intra-blocks, also referred to as intra-block refresh. By intentially turning off the motion compensation for some macroblocks, the intra-macro blocks do not depend on past frames and error propagation can be minimized for those blocks. In “Optimal intra coding of blocks for robust video communication over the internet,” Image Communication, September 1999, Cote et al. described a random intra-block refresh technique to reduce the error propagation. The intra-block error rate β, influences both bit rate and the transmission distortion.
The above prior art techniques for error-resilient video can be classified into two main categories depending on the means of adding redundancy. The redundancy can be added explicitly, e.g., FEC, or implicitly, e.g., intra-block refresh. However, it should be noted that different error resilient tools differ in the methods for adding redundancy and utilizing redundancy to reduce distortions, i.e., it should be possible to provide a better error resilience tool to further reduce distortion with the same amount of redundancy.
Another important problem in designing an error resilient video encoding scheme is to determine the amount of redundancy to add. Ideally, this should change adaptively according to current video signal and current channel characteristics. If the channel has a smaller error rate, or if the frames contain less motion, then less redundancy is needed. Otherwise, more redundancy should be added. Usually, because the signal and channel characteristics vary over time, a good video encoding scheme for transmission should also be changing the coding methodology and the amount of the redundancy dynamically based on the signal and channel characteristics.
In U.S. Pat. No. 6,339,450, “Error resilient transcoding for video over wireless channels,” issued on Jan. 15, 2002 to Chang et al., an error resilient transcoder is described. The transcoder maintains the resilience at an optimal level to accommodate the prevailing channel characteristics as measured by the BER of a wireless channel. The error resilience tools used include means for increasing the proportion of intra-blocks and increasing the length of slices.
In “Joint Source Channel Rate-Distortion Analysis for Adaptive Mode Selection and Rate Control in Wireless Video Coding” IEEE Trans. Circuits and System for Video Technology, June 2002, He et al. described a scheme to adaptively select intra-block rate and source-channel bit allocation based on time-varying wireless channel characteristics.
FIG. 1 shows the general structure of most prior art error-resilient encoders. Input video 101 is fed into a single description, error-resilient video encoder 110. The encoder 110 includes error resilience tools such as intra-block refresh. A statistics generator 120 extracts source statistics 121. A rate-distortion (R-D) statistics analyzer 130 adjusts the encoding behavior according to the source statistics 121 and channel characteristics 151. The R-D statistics 131 are used by a parameter and method selector 140 to determine coding parameters and coding methods 141, which in turn control the output 111 of the encoder 110.
To calculate the end-to-end distortion, which includes the distortion caused by transmission errors, the R-D statistics analyzer 130 requires the characteristics 151 of a transmission channel 150, such as packet loss rate 152. In addition, the channel throughput 153 also keeps changing. Therefore, the throughput is sent to the parameter and method selector 140 to help the process of parameter and method selection. Using the selected coding methods and parameters, the encoder 110 encodes the input signal 101 and outputs a coded bitstream 111 based on a single description.
Most prior art video coding techniques utilizing the above structure generate one bitstream for a video signal. Such coding schemes are referred to as single description error-resilient encoders and the bitstream produced by the encoder is referred to as single description. The advantage of single description coding is that the receiver can decode the video signal at a highest possible quality when the bitstream is received completely. However, if parts of the stream cannot be received, then received parts are of little use.
Multiple description coding (MDC) is different from traditional coding techniques. With MDC, a video signal is encoded into several bitstreams, i.e., descriptions. Now the receiver can recover the encoded signal by receiving any of these descriptions. In general, the more descriptions that are received, a higher quality decoded video is obtained. The motivation for using MDC is because the probability of losing all streams is much lower than losing one stream. Consequently, the receiver has a much higher chance to reconstruct the video with acceptable quality compared to using single description coding.
FIGS. 2A and 2B shows the basic structure of a prior art MD video encoder 210 and decoder 220 based on two descriptions. The MDC encoder 210 generates two bitstreams 211–212 from an input video 201 using a fixed set of parameters 202 and transmits the stream over two channels 213–214. The decoder 220 has three decoders 222–224, which are used dynamically according to the information received. If both descriptions, i.e., bitstreams, are received, then decoder II 223 is used to output a signal 226 with a lowest distortion. If only one bitstream, e.g., bitstream A 211, is received, decoder I 222 is used to output a signal 225 with a higher distortion. The same is true if only bitstream B 212 is received. In that case, decoder III 224 is used to output a signal 227 with a higher distortion.
FIG. 2C shows two example bitstreams 211 and 212 generated by a prior art MD video encoder. Bitstream 211 has the bits of coded even frames (2, 4, 6). Bitstreams 212 has the bits of coded odd frames (1, 3, 5). For each coded frame, there are two parts: a central part and a side part.
An MDC video coding scheme “Multiple Description Motion Compensation (MDMC)” was described by Wang et al. in “Error Resilient Video Coding Using Multiple Description Motion Compensation,” IEEE Transaction on Circuits and Systems for Video Technology, June 2002.
FIG. 3A shows the encoder of that MDMC scheme 300. There are three primary components: a central encoder 310, an even side encoder 320 and an odd side encoder 330.
All frames of an input video 301 are fed into the central encoder 310. In the central encoder, a central prediction frame 351, c_pred, is constructed by a linear combination of two prediction frames 371–372, pred_frame1 and pred_frame2, as:c_pred=a1*pred_frame1+(1−a1)*pred_frame2,  (1)where a1 is the weighting parameter used by a central predictor 350.
A difference 305 between the current frame and the prediction frame 351 is subject to a DCT 340, and then quantized by the central quantizer 345 with quantization parameter, Q0. The quantized DCT coefficients are variable length coded (VLC) 375 and sent as central part of bitstream A 376 when the current frame is an even frame, and as central part of bitstream B 377 when the current frame is an odd frame.
A frame reconstructor 355 inversely quantizes the quantized DCT coefficients, performs an inverse DCT, and then adds back the central prediction frame. The result is stored in a frame memory 360, which also stores two previously encoded frames.
A motion estimator 365 carries out motion estimation and determines two motion vectors, one between the current frame and previously coded even frame, and the other one between the current frame and previously coded odd frame. The two motion vectors and the two coded frames are sent into a prediction frame constructor 370, and two prediction frames, pred_frame1 373 and pred_frame2 374, are generated. Those two prediction frames are passed to the central predictor 350 and the central prediction frame 351 is constructed by a linear combination of the two prediction frames.
The even frames 302 of the input video sequence 301 are fed into the even side encoder 320. In the even side encoder, a side predictor 321 generates an even frame prediction frame 322 using pred_frame2 372. A difference 323 between the current frame and the prediction frame is DCT quantized 324. The DCT coefficients are subtracted 325 from the quantized DCT coefficients of the central branch, and then a side quantizer 326 quantizes the resulting signal with a quantization parameter, Q1. The quantized signal is variable length coding (VLC) 327 coded and sent as the side part of bitstream A 328.
Odd frames 303 of the input video sequence 301 are sent into the third branch, which is symmetric to the second branch and is illustrated simply as a side encoder for odd frames 330. The generated bitstream is sent as the side part of bitstream B 331.
In a video transmission system, the thus generated bitstreams are sent over different channels, which may either be two separate physical paths or a single path with two virtual channels. Assuming two bitstreams and two channels, there are three possible scenarios in the receiver. In the following, the decoding is described assuming frame level processing, however macroblock level processing can also be applied.
In the first scenario, as shown in FIG. 3B, the information for both the current frame, e.g., bitstream A, and the corresponding previous odd frame, bitstream B, are received. In that case, a conventional decoding strategy is applied. Bitstream A is first variable length decoded 378, then the bitstream is dequantized 380 and subject to an inverse DCT 382. The resulting signal is then added 383 to the prediction frame 389 generated by central predictor 388. The reconstructed frame 384 is finally displayed 385 and stored in the Frame Memory 386.
In the second scenario, as shown in FIG. 3C, the information of the current frame, e.g., bitstream A, is received and the corresponding previous odd frame, bitstream B, is lost. In this case, the side decoder is used. Both the central and side parts of bitstream A are passed through their respective VLD and dequantization processes. The two resulting components are then summed. This signal is then subject to an inverse DCT and added to the even prediction frame, which was constructed only from the previous even frame by the side predictor and prediction frame constructor. The resulting frame from the summation is the reconstructed even frame.
Because the previous odd frame is lost and current reconstructed even frame contains some information of the previous odd frame, a lost frame estimator or an odd frame estimator can be used to construct an odd frame to be used as a reference frame. After that, the estimated odd frame and the reconstructed even frame are displayed and stored in the frame memory 386.
In the third scenario, the information of the current frame is lost. Then, the decoder must wait for the next frame. If the next frame is received, then the side decoding strategy described for the second scenario is used. If the next frame is also lost, then a conventional error concealment method such as copying the previous frame can be used.
From the above scenarios, it is evident that if both streams are received, then the side part of the bitstream is totally useless. On the other hand, if some error happens, the side part of a bitstream can help to reconstruct a frame with lower, but still acceptable quality. So, the side part of a bitstream is considered redundant for the purpose of error resilience. It should be noted that the amount of the redundancy can be controlled by Q1, the quantization parameter used by the side quantizer.
The coding efficiency of both the central encoder and side encoder rely on the choice of Q0, the central quantization parameter, and a1, the weight used by the central predictor. Both of these parameters have a direct influence on the redundancy introduced since the mismatch signal depends on the output of the central quantizer, as well as the central prediction frame that is used to produce the input to the central quantizer.
Prior art adaptive error-resilient video coding focuses on single description codecs. However, the existing error-resilience models and techniques used by these codecs are not easily extended to multiple description coding. Also, the redundancy control methods are incompatible with each other. It is important for an error-resilient video coding scheme to be adaptive to changing signal and channel characteristics.
The performance of an MDMC codec, namely the coding efficiency and error resilience, is controlled by four parameters, a1, Q0, Q1, and β. In the prior art, the set of coding parameters used for coding are selected heuristically and implemented for fixed periods without regard for changing signal and channel characteristics. That is less than optimal. Therefore, there is a need for providing an adaptive MDMC coder that can select the coding methods and parameters dynamically based on the signal and channel characteristics.